Hyperplane \ Spin " Dynamics , NetworkPlasticity and Back - Propagation Learning

The processing performed by a feed-forward neural network is often interpreted through use of decision hyperplanes at each layer. The adaptation process, however, is normally explained using an error landscape picture. In this paper the actual dynamics of the decision hyperplanes is investigated. As a result a mechanical analogy is drawn with a system of spins acted upon by forces. The spin objects have a variable mass, and relaxation in the system is represented by increased overall system mass, also termed \cooling". The analogy is used to clarify the dynamics of learning, information storage and ro-bustness, and in particular the functioning of the process of back-propagation. Learning deadlocks and local minima are explained. The concept of network \plasticity" is introduced, and used to understand destructive relearning, and how it may thus be better avoided. Practical beneets from this interpretation are illustrated with hints for optimal weight initializations, avoidance of and escape from some types of local minima, and avoidance of destructive relearning through selective reordering of some types of networks.